Preferential selection of candidates for delta compression

ABSTRACT

A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and generates a sketch for the selected data chunk. The method and system search for a set of candidate data chunks with a matching sketch and rank the set of candidate data chunks by degree of sketch matching. The set of candidate data chunks are tie-braked using location status data for each candidate and the selected data chunk is delta compressed with a selected candidate data chunk. The delta compressed selected data chunk is then stored in a data storage system.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/495,831 filed Jun. 13, 2012, entitled “PREFERENTIAL SELECTION OFCANDIDATES FOR DELTA COMPRESSION.” This application is related to aco-pending application entitled “PREFETCH DATA NEEDED IN THE NEAR FUTUREFOR DELTA COMPRESSION,” filed Jun. 13, 2012 (Now U.S. application Ser.No. 13/495,795). This application is related to a co-pending applicationentitled “METHOD FOR CLEANING A DELTA STORAGE SYSTEM,” filed Jun. 13,2012 (Now U.S. application Ser. No. 13/495,881). This application isrelated to a co-pending application entitled “DEDUPLICATION USINGSUB-CHUNK FINGERPRINTS,” filed Jun. 13, 2012 (Now. U.S. application Ser.No. 13/495,856). This application is related to a co-pending applicationentitled “PREFERENTIAL SELECTION OF CANDIDATES FOR DELTA COMPRESSION,”filed Jun. 13, 2012 (Now U.S. application Ser. No. 13/495,859). Thisapplication is related to U.S. application Ser. No. 13/495,868 entitled“PREFERENTIAL SELECTION OF CANDIDATES FOR DELTA COMPRESSION,” filed Jun.13, 2012 (Now U.S. Pat. No. 8,712,978). This application is related to aco-pending application entitled “METHOD FOR CLEANING A DELTA STORAGESYSTEM,” filed Jun. 13, 2012 (Now U.S. application Ser. No. 13/495,893).This application is related to a co-pending application entitled “METHODFOR CLEANING A DELTA STORAGE SYSTEM,” filed Jun. 13, 2012 (Now U.S.application Ser. No. 13/495,926).

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to data storagesystems. More particularly, embodiments of the invention relate tomethods and systems for improved delta compression management.

BACKGROUND

Many companies and individuals with large amounts of stored data employa backup data storage system. These backup data storage systems can belocated local to the data to be backed up or at a remote site. Thebackup data storage systems can be managed by the entity controlling theprimary data storage devices or a backup data storage service company.Data can be backed up at any frequency and any amount of data can bebacked up. The backed up data can be retrieved in the case of a failureof a primary storage device from the backup data storage system. Wherethe backup is managed by a service company the data of many separateclients of the backup service can be backed up into the backup datastorage system.

Compression of data before storage is widespread within the backupstorage service market. Delta compression is a particular type ofcompression that is applied in this field. The basic idea of deltacompression is to divide incoming data into smaller units called datachunks, generate a resemblance hash over the data chunk (this hashresult is referred to herein as a “sketch”), and check the sketchagainst an index of previously stored data chunks. A sketch can begenerated by identifying “features” of a data chunk that would notlikely change even as small variations are introduced into the datachunk. In one example embodiment, a rolling hash function (e.g., togenerate a Rabin fingerprint) is applied over all overlapping smallregions of the data chunk (e.g., a 32 byte window) and the features areselected from maximal hash values generated in the process. This can bedone with multiple hash functions to generate any number of features(e.g., generating four features). These features can be furtherprocessed to generate “super features,” which can be generated by theapplication of a hashing function over the feature values. Whilefeatures are described herein as being used for similarity matching, oneskilled in the art would understand that super features can also be usedfor this purpose and are considered herein as a subset of the use offeatures and feature matching.

Data chunks with identical or similar sketches are considered for deltacompression, while sketches that are not found in the index cause thecorresponding data chunk to be stored and the sketch added to the index.In this way only a sub-set of the data chunks need to be stored in theirentirety. For each data chunk with a matched sketch the system encodes adata chunk as a reference to the similar data chunk (i.e., the datachunk with the matching sketch) and a difference or change in the datachunk being compressed relative to the similar data chunk. In this way,only the original data chunk and a difference (i.e., the delta) betweenthe two similar data chunks are stored rather than two entire datachunks, thereby reducing the amount of data that must be storedsignificantly.

Each of the data chunks also has a secured hash executed over it togenerate a result that is referred to herein as a “fingerprint.” Thefingerprint can be utilized to identify a specific data chunk. The datachunks can be portions of a file. The file has a recipe forreconstruction, which consists of a list of fingerprints and relatedinformation corresponding to the chunks stored in the backup datastorage system. For backup data storage systems, the typical backupcycle consists of daily or weekly full backups of a primary data storagesystem, so most of the data stored in the backup data storage system isrepeated (i.e., a duplicate) or has minor alterations. These minoralterations can be effectively delta compressed to avoid having to storesignificant amounts of data.

Delta compression also has applicability to data transmission whereinstead of minimizing an amount of data storage space that is required,the delta compression seeks to minimize the amount of data that must betransmitted thereby reducing bandwidth requirements. The delta compresstransmission process works in a similar manner to that of the deltacompression for storage, except that instead of storing the deltacompressed data, the delta compressed data is transmitted to a receivingcomputer. Fingerprint data is also transmitted to the remote machinesuch that a fingerprint index can be maintained at the receiving machineto enable proper reconstruction of the transmitted data at the receivingmachine.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a block diagram of one embodiment of a delta compressionsystem.

FIG. 2 is a flowchart of one embodiment of a delta compression process.

FIG. 3 is a flowchart of one embodiment of a garbage collection process.

FIG. 4 is a diagram of one embodiment of a garbage collection processwhere references to base chunks are maintained in the file description.

FIG. 5 is a diagram of one embodiment of a garbage collection processwhere references to base chunks are maintained in the metadata of a datachunk.

FIG. 6 is a diagram of one embodiment of a garbage collection processwhere references to base chunks are maintained in an index.

FIG. 7 is a flowchart of one embodiment of a garbage collection processwhere references to base chunks are maintained in the file description.

FIG. 8 is a flowchart of one embodiment of a garbage collection processwhere references to base chunks are maintained in the metadata of a datachunk.

FIG. 9 is a flowchart of one embodiment of a garbage collection processwhere references to base chunks are maintained in the index.

FIG. 10 is a flowchart of one embodiment of a sanitation process.

FIG. 11 is a flowchart of one embodiment of a data chunk pre-fetchingprocess.

FIG. 12 is a flowchart of one embodiment of a data chunk selectionprocess where candidates are ranked by sketches.

FIG. 13 is a flowchart of one embodiment of a data chunk selectionprocess where candidates are ranked by location status.

FIG. 14 is a flowchart of one embodiment of a process for dynamic datachunk selection process.

FIG. 15 is a flowchart of one embodiment of a process for sub-chunk datadeduplication.

FIG. 16 is a diagram of one embodiment of a stage of a datadeduplication process with sub-chunks.

FIG. 17 is a diagram of one embodiment of a chunk encoding withsub-chunk data deduplication.

DETAILED DESCRIPTION

Various embodiments and aspects of the inventions will be described withreference to details discussed below, and the accompanying drawings willillustrate the various embodiments. The following description anddrawings are illustrative of the invention and are not to be construedas limiting the invention. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentinvention. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present invention.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment.

In one embodiment, a delta compression system is a data storage systemhaving a set of storage units. A ‘set,’ as used herein refers to anypositive whole number of items including one item. Each of the storageunits is configured to store data chunks that are delta compressedwithin each of the data storage units, where the files stored in thestorage system are comprised of the data chunks. Each data chunk isformed through a specific policy, e.g. by breaking a file up based onproperties of the content in the file, which permits changes to one datachunk without affecting the boundaries or contents of other chunks. Eachdata chunk is associated with a ‘fingerprint’ and similar metadatarepresenting the data chunk. A fingerprint contains a metadata value(e.g., a hash value) that uniquely identifies that data object.Fingerprints can be used to identify duplicate data chunks. As usedherein, this metadata is in contrast with the ‘content’ of the data inthe files. Similarly, each data chunk is associated with a ‘sketch’which is a metadata value that is generated by a resemblance hash orsimilar algorithm to be used to identify similar data chunks.

In other embodiments, the delta compression system is utilized toimprove the efficiency of data transfer by reducing the amount of datato be transmitted and thereby reduce the bandwidth utilization orrequirements of the data transfer. In these embodiments, the metadatanecessary for decompressing the transmitted data is also transmitted tothe recipient device along with the compressed data chunks. This caninclude fingerprint information and similar information to enable theconstruction of an index at the recipient device to identify base chunksreferenced by delta chunks and similar relationships necessary forreconstruction of a data set. A ‘base chunk,’ as used herein refers to adata chunk that is utilized as a baseline for delta compression, wherethe difference between the data chunk serving as the base chunk and thedata chunk being delta compressed is referred to herein as the ‘delta.’The delta can be stored in place of the complete data chunk along with a‘delta reference’ that identifies the base chunk.

The system and processes described herein efficiently delta compress aset of data chunks and manage the use of the set of data. Theseprocesses and systems include garbage collection, cache utilization andpre-fetching processes and systems. These processes and systems can beutilized separately and in any combination with one another. The deltacompression processes and systems can be utilized in combination withother data storage and transmission systems and processes such as datadeduplication processes and systems. One skilled in the art wouldunderstand that the delta compression processes and systems are notdependent on these additional systems and can be utilized without them.Rather, other systems are presented by way of example and not by way oflimitation to demonstrate the inter-compatibility of the deltacompression processes and systems.

FIG. 1 is a diagram of one embodiment of a data storage system and anetwork of associated machines. Referring to FIG. 1, the system 100includes, but is not limited to, one or more client systems 101, 102communicatively coupled to delta compression system 104 over network103. Clients 101, 102 can be any type of computing devices such as aserver, a personal computer (e.g., desktops, laptops, and tablets), a“thin” client, a personal digital assistant (PDA), a Web enabledappliance, a gaming device, a media player, or a mobile phone (e.g.,Smartphone), or similar computing device. Network 103 can be any type ofnetwork such as a local area network (LAN), a wide area network (WAN)such as the Internet, a corporate intranet, a metropolitan area network(MAN), a storage area network (SAN), a bus, or a combination thereof,where constituent devices within the networks communicate over anycombination of wired and/or wireless transport mediums. The clients 101,102 transmit data to or receive data from the delta compression system104 over the network 103.

Delta compression system 104 can include any type of individualcomputing device, server or cluster of servers. For example, deltacompression system 104 can be a storage server used for any of variousdifferent purposes, such as to provide multiple users with access toshared data and/or to back up mission critical data. In another example,the delta compression system can be a file server that compresses largefiles for transmission to clients 101, 102. Delta compression system 104can be, for example, a file server (e.g., an appliance used to providenetwork attached storage (NAS) capability), a block-based data storageserver (e.g., used to provide storage area network (SAN) capability), aunified data storage device (e.g., one which combines NAS and SANcapabilities), a nearline data storage device, a direct attached storage(DAS) device, a tape backup device, or any other type of data storagedevice. Delta compression system 104 can have a distributedarchitecture, or all of its components can be integrated into a singleunit. In one embodiment, delta compression system 104 can be implementedas part of an archive and/or backup system such as a data storage systemavailable from EMC® Corporation of Hopkinton, Mass.

In one embodiment, delta compression system 104 includes, but is notlimited to including a set of data storage units 108, 109, 110,processing system 151, working memory 106, interconnect 120, interface131 and similar components. The processing system 151 can include a setof processing devices for processing and managing data within the deltacompression system 104. The working memory 106 is used by the processingsystem 151 to manipulate data and execute instructions. In oneembodiment, the processing system 151 executes instructions to implementthe garbage collection module 161, deduplication module 163,sanitization module 165, pre-fetch module 167, delta processing module169, data chunk storage engine 107, preferential selection module 173and similar components and uses the working memory 106 in this process.The working memory 106 can include a cache 181 to store frequently useddata, an index 183 or portion of an index that correlates data chunks tostorage locations and a queue 185 that can be utilized to store datachunks or references to data chunks stored in other locations that arescheduled for delta compression processing or similar functions.

The garbage collection module 161, deduplication module 163,sanitization module 165, pre-fetch module 167, delta processing module169, data chunk storage engine 107, preferential selection module 173,data storage units 108, 109, 110, and similar components arecommunicatively coupled to one another in the processing system 151 andworking memory 106. Data storage units 108, 109, 110 can be implementedlocally (e.g., single node operating environment) or remotely (e.g.,multi-node operating environment) via interconnect 120, which may be abus or a network.

A delta compression module 169 receives or retrieves a set of datachunks that are to be delta compressed. The delta compression process isgenerally described below in regard to FIG. 2. The delta compressionmodule 169 identifies data chunks by generating a fingerprint and/or asketch for each data chunk as it is being processed (if not alreadycreated by other components). A data chunk being processed in thismanner is referred to herein as a selected data chunk, in that the datachunk has been selected to be processed. The sketch is matched againstsketches of previously processed data chunks using an index 129 orsimilar metadata source. A data chunk with a matching sketch or similarsketch to the sketch of the selected data chunk can be selected to serveas a ‘base’ chunk. The base chunk is already stored and will bemaintained for reference. The selected data chunk is then reduced to arepresentation of the differences between the selected data chunk andthe base chunk. This difference or ‘delta’ is stored along with areference to the base chunk, which is referred to herein as a deltareference. Storing and/or transmitting a delta and a delta referencerequire fewer resources than storing or transmitting the entire datachunk.

The selection of a base chunk can be influenced by more than just amatching sketch. A preferential selection module 173 can work incoordination with the delta processing module 169 to select a basechunk. The criteria for selecting the base chunk can include both thesimilarity of the base chunk to the selected data chunk (e.g., based onthe number of matching features between the sketches) and the currentlocation and status of the base chunk. The base chunk could be locatedin different storage locations that have varying access times. Forexample the base chunk could be in a cache 181 in the working memory orin the data chunk areas 112, 113 of the data storage unit 108, 109, 110.The access times for a cache can be significantly less than those of adata storage unit 108, 109, 110. The base chunk could also be in acompressed state or other state that requires processing to access thebase chunk. The preferential selection module 173 identifies a set ofcandidate data chunks and based on its configuration selects a basechunk from this set of candidate data chunks in light of access timesand desired level of compression. This process is discussed in furtherdetail herein below in regard to FIGS. 12-14.

A pre-fetch module 167 seeks to load data chunks into a cache 181,working memory 106 or similar location with faster access times inanticipation that these data chunks are likely to be utilized by otherprocesses such as the delta compression process or the deduplicationprocess. The pre-fetch module 167 analyzes selected data chunks and basechunks and attempts to identify similar data chunks to load into thecache 181. The pre-fetch module 167 can identify a set of similaritydata using various methods including sketches, addresses, containers andsimilar metadata related to selected data chunks and/or base chunks. Thepre-fetch module 167 then seeks to load those data chunks in datastorage units 108, 109, 110 or similar locations with slow access timesinto a cache 181 or similar location with a faster access time. Thepre-fetch process is described in further detail herein below in regardto FIG. 11.

A deduplication module 163 can work in coordination with othercomponents to reduce the number of data chunks that need to be stored ortransmitted. The deduplication module finds duplicate data chunks byusing fingerprints of each data chunks to identify duplicate datachunks. These duplicate data chunks can then be removed to reducestorage and bandwidth requirements. The deduplication module 163 canalso execute the sub-chunk deduplication process described herein belowwith regard to FIGS. 15-17. In other embodiments, the sub-chunkdeduplication process can be executed by the delta compression module169, other modules or any combination thereof.

The garbage collection module 161 traverses the data storage units 108,109, 110 to find ‘dead’ data chunks that are data chunks that are nolonger referenced by any file recipes in the delta compression system104 or by any data chunks that have been delta compressed. The garbagecollection process is discussed in further detail herein below inreference to FIGS. 3-9. The garbage collection module 161 can runasynchronous to any delta-compression processes or deduplicationprocesses and preserves the data that is necessary for their efficientand correct operation.

A sanitization module 165 can be a component of the garbage collectionmodule 161 or a separate component. The sanitization module 165 ensuresthat sensitive data is deleted when intended and that the deltacompression does not preserve sensitive data that is designated fordeletion. The process identifies each data chunk of a file that isdesignated for deletion. If some of these data chunks serve as basechunks for other delta encoded data chunks then under normalcircumstances these data chunks may not be deleted. However, due torequirements for the complete deletion of some sensitive files, thesanitization module 165 identifies the delta referencing data chunks anddecodes them or alters their delta references to a new base chunkthereby allowing deletion of the data chunk of the file to be deletedwithout causing any loss of data for those referencing delta encodeddata chunks. This process is described in further detail in reference toFIG. 10.

In one embodiment, one of the data storage units 108, 109, 110 operatesas an active storage to receive and store external or recently receiveduser data, while the other data storage unit operates as a secondarydata storage unit to periodically archive data from the active storageunit according to an archiving policy or scheme. Data storage units 108,109, 110 can be, for example, conventional magnetic disks, optical diskssuch as CD-ROM or DVD based storage, magnetic tape storage,magneto-optical (MO) storage media, solid state disks, flash memorybased devices, or any other type of non-volatile storage devicessuitable for storing large volumes of data. Data storage units 108, 109,110 can also be combinations of such devices. In the case of diskstorage media, the data storage units 108, 109, 110 can be organizedinto one or more volumes of Redundant Array of Inexpensive Disks(RAIDs).

In response to a data file being received to be stored in data storageunits 108, 109, 110 data chunk storage engine 107 is configured todivide (or segment) the data file into multiple data chunks according toa variety of data division policies or rules. In one embodiment, datachunk storage engine 107 in coordination with the deduplication module163 and delta processing module 169 only stores a chunk in a datastorage unit if the data chunk has not been previously stored in thestorage unit. In the event that a data chunk has been previously stored,metadata information is stored enabling the reconstruction of a fileusing the previously stored data chunk. As a result, chunks of datafiles are stored in a deduplicated and/or delta compressed manner,either within each of data storage units 108, 109, 110 or across atleast some of the data storage units 108, 109, 110. In addition,metadata is added to at least some of data storage units 108, 109, 110such that files can be accessed independent of another storage unit.Data stored in the data storage units can be stored in a compressed form(e.g., lossless compression: Huffman coding, Lempel-Ziv Welch coding;delta encoding, i.e., a reference to a data chunk plus a difference (a‘delta’)). In one embodiment, different data storage units can usedifferent compressions (e.g., main or active data storage unit 108 usinga different compression from other data storage units, one storage unitfrom another storage unit, or similar variations). Also, differentchunks can use different compressions in any combination orconfiguration.

Data chunks and metadata can also be further organized into a set ofcontainers. Containers are logical units of storage within the physicaldata storage units 108, 109, 110. Containers can be structured such thatentire containers of data chunks and/or meta data can be retrieved atone time and stored in working memory or similar faster access location.

In a further embodiment, the data chunks can be further sub-divided intosub-chunks. Sub-chunk deduplication can then be employed to furtherreduce duplication by dividing data into smaller chunks. The principlesand features described herein in relation to chunks would be understoodby one skilled in the art to be generally applicable to sub-chunks aswell.

In one embodiment, the metadata 114, 115, 127 information includes afile name, a storage unit where the data chunks associated with the filename are stored, reconstruction information for the file using the datachunks, data chunk type, local compression size, sub-chunk sizes,fingerprint, data chunk hashes, data chunk resemblance hashes for deltaencoding, disk location, sketches, container information and any otherappropriate metadata information. In one embodiment, a copy of themetadata is stored on a particular storage unit for files stored on thatstorage unit so that files that are stored on the storage unit can beaccessed using only the metadata stored on the data storage unit. In oneembodiment, a main set of metadata information can be reconstructed byusing information of other data storage units associated with thestorage system in the event that the main metadata is lost, corrupted,damaged, or similarly unavailable. Metadata for a data storage unit canbe reconstructed using metadata information stored on an active datastorage unit or other storage unit (e.g., replica storage unit).

Metadata information further includes index information 129 (e.g.,location information for chunks in storage units). The index can includeentries for each data chunk and/or sub-chunk stored or managed by thedelta compression system 104. Portions 183 of the index information 129can be loaded into the working memory that are related to a set of datachunks being processed. The index 129 is generally too large to beentirely loaded into the working memory 106. The index 129 can bepartially loaded on a container by container basis or similar groupingof index entries. The loading of this index information can be handledin a manner similar to that of the corresponding data chunks with regardto pre-fetching processes.

In one embodiment, wherein the delta compression system serves as a datastorage system, the data storage system can be used as a tier of storagein a storage hierarchy that comprises other tiers of storage. One ormore tiers of storage in this hierarchy can utilize different kinds ofstorage devices and/or can be optimized for different characteristicssuch as random update performance. Files are periodically moved amongthe tiers based on data management policies to achieve a cost-effectivematch to the current storage requirements of the files. For example, afile can initially be stored in a tier of storage that offers highperformance for reads and writes. As the file ages, it may be moved intoa tier of storage according to this invention. In various embodiments,tiers include different storage technologies (e.g., tape, hard drives,semiconductor-based memories, optical drives, etc.), different locations(e.g., local computer storage, local network storage, remote networkstorage, distributed storage, cloud storage, archive storage, vaultstorage, etc.), or any other appropriate storage for a tiered datastorage system.

Metadata 114, 115 and 127 can include a fingerprint and/or a sketch of adata chunk. In one embodiment, a fingerprint may include a hash valuecomputed based on the data chunk using a variety of hash algorithms,such as SHA-1 or message digest 5 (MD5) hash algorithms. Alternatively,metadata can include a sketch determined from the corresponding datachunk using any one of a variety of methods. Sketch generation involvesgenerating a value from a data chunk that is an approximate identifierof the data chunk such that the value would also identify similar datachunks. The sketches can be generated by use of a resemblance hashfunction or similar function.

Note that throughout this application, a data storage system is utilizedas an example application of a delta compression system. However, thetechniques described throughout this application can also be applied toother types of system including those that utilize delta compression fordata transmission. Also note that fingerprints and sketches are utilizedas examples of metadata, however, other types of data such as a feature,signature, or pattern of data can also be applied. If data aretransformed by filtering portions of content, the transformed data andmetadata therein will be distinct from the fingerprint used for deltacompression and will be computed and stored separately.

In one embodiment, a given data storage unit 110 can store file recipedata 129, metadata 127 and an index 129. In other embodiments, this datais distributed over all of the data storage units 108, 109, 110. Thefile recipe data 125 contains the information necessary to reconstruct astored file in the data storage units 108, 109. The file recipe 125includes fingerprints or similar identifiers of each of the data chunksthat make up the file. These can be used to retrieve or generate thecorresponding data chunk to form the file. The file recipe 125 can alsoinclude additional data including delta referencing data that identifiesthe data chunks that other data chunks rely upon as base chunks fortheir delta compression.

The index 129 is a data structure that correlates data chunks with theirlocation in the data storage units 108, 109. Each index entry can alsoinclude metadata about the data chunks including their fingerprints,sketches, delta references, container information and similarinformation. The index 129 is utilized to locate data chunks in the datastorage units 108, 109. The data chunks 112, 113 can be organized in anymanner within the data storage units 108, 109. In one embodiment, thedata chunks 112, 113 are organized in sets referred to as containers.Metadata for the data chunks can then be container specific. Data chunks112, 113 can be organized into containers or similar structures based onsimilarity, stream order, file organization or similar criteria.

FIG. 2 is a flowchart of one embodiment of a delta compression process.In one embodiment, the delta compression process is initiated inresponse receiving data to be compressed (Block 201). The data can bereceived from external or internal sources for backup, transmission, orsimilar compression scenarios. In other embodiments, the deltacompression is initiated and the process is directed to or accesseslocal or remote data to be processed. The data set is divided into a setof data chunks (Block 203). The data chunks can have a size such as 4 kbor 8 kb as designated by an administrator, configuration information ordefault settings. In one embodiment, the data chunks have a fixedregular and uniform size. In other embodiments, the data chunks can havevariable sizes.

For each data chunk in the set of data a fingerprint is calculated(Block 205). A fingerprint can be generated by a secure hashingalgorithm or similar function such that the fingerprint uniquelyidentifies the data chunk. The fingerprint can have a size proportionateto the number of unique data chunks that must be identified and theacceptable probability that a collision occurs where two different datachunks have the same fingerprint, for example the fingerprint could be20 bytes for a 4 kb data chunk within a petabyte storage system.

The fingerprints can be utilized to deduplicate the data chunks in theset of data (Block 207). Each data chunk with an identical fingerprintto a previously processed data chunk can be discarded and a reference tothe other data chunk utilized in its place in the storage ortransmission of the data set. This enables a single instance of a datachunk to represent all other instances of that data chunk in the dataset providing considerable resource savings in space or bandwidth. Thededuplication process is optional in a delta compression system.

The delta compression continues by selecting a next chunk in a stream orqueue of data chunks to process serially or in parallel (Block 209). Theselected data chunk is analyzed to generate similarity information thatwill enable the identification of other data chunks with similar data.In one embodiment, this is accomplished by generating a sketch using aresemblance hash or similar function executed over the data chunk (Block211). The sketch is utilized to identify a set of candidate chunks toserve as a base chunk for the currently selected chunk (Block 213). Thesketch can be utilized to find matching or similar sketches associatedwith previously processed data chunks. An optimum data chunk can then beselected from these candidates to utilize as a base chunk in performinga delta compression of the selected data chunk (Block 215). The basechunk is then read from its storage location such as the cache, memoryor data storage (Block 217).

The base chunk is compared with the selected chunk to identifydifferences in the data content and data organization. This differing or‘delta’ information is preserved, or a means of recreating thesedifferences based on modifying the base chunk is preserved, as deltainformation (Block 219). The delta information is then stored along witha reference to the base chunk to enable decoding and reconstruction ofthe selected data chunk (Block 221).

This process is provided by way of example. The processes that followseek to improve upon the basic process for delta processing. One skilledin the art would understand that any one of or any combination of theprocesses described herein could be utilized to improve the efficiency,compression and/or resource usage of the delta compression system overthe basic delta compression process.

FIG. 3 is a flowchart of one embodiment of a garbage collection process.In one embodiment, the garbage collection process can be initiated atany time asynchronous to any delta compression process (Block 301). Thegarbage collection process can be executed on a set of data storageunits storing a set of data chunks in containers or similar datastructures. These data chunks can be part of a data backup system, datatransmission system or similar data management system where deltacompression is applicable. The garbage collection process can be runduring offline times or low system usage times for the delta compressionsystem to minimize any impact on the usage of the delta compressionsystem. In other embodiments, separate hardware or machines execute thegarbage collection process to minimize the impact on the hardware ormachines utilized by the delta compression system.

The garbage collection process traverses the data structure containingthe data chunks themselves or the metadata of the data chunks toidentify each live data chunk that is referenced by a valid or activefile managed by the delta compression system (Block 303). In oneembodiment, the files are tracked as a set of file recipes, whichidentify each of the data chunks that constitute a particular file.Thus, each file recipe is traversed and each of the data chunksreferenced by the file recipe is added to a live chunk list.

Some of the data chunks that make up each of these files can be storedas a delta encoded data chunk. For each data chunk in the file recipethat refers to a delta encoded data chunk, the corresponding base chunkmust also be preserved and is added to the live chunk list. The basechunk can be determined by looking up a delta reference in the filerecipe, file metadata or the index. Each of these scenarios is discussedfurther herein below in relation to FIGS. 4-6.

Optionally each live data chunk that is repeated in the live chunk listcan be deduplicated through a deduplication process that replacesduplicate live chunks with references to a single live chunk that ismaintained (Block 307). In one embodiment, live base versions of theduplicate data chunks are maintained while delta encoded versions arediscarded. In other embodiments, the deduplication process if utilizedis asynchronous to the garbage collection process.

Once all of the file recipes or similar file information is traversed,then the data chunks in the live chunk list are maintained and otherdata chunks are discarded (Block 309). This process can be effected bycopying the live data chunks to a new location, deleting the discardeddata chunks in a current location or similarly reclaiming the resourcesof the ‘dead’ data chunks that did not make the live chunk list. Thisdecision can be dynamic based on a condition of the data structurecontaining the data chunks such as a percentage of a container that islive where a low percentage would result in a copy of the live chunksand a high percentage would result in a deletion of the dead datachunks.

In one embodiment, the garbage collection process can be a part of agenerational process where the set of data chunks is stored andmaintained as a ‘cliff’ or differential encoding of a prior version ofthe set of data chunks. In this embodiment, the garbage collectionprocess must be careful to add base chunks to a live chunk list that maybe in a prior version data set, making the garbage collection processcompatible with the generational process for managing the data set.

FIG. 4 is a diagram of one embodiment of a garbage collection processwhere references to base chunks are maintained in the file description.FIG. 7 is a flowchart of one embodiment of a garbage collection processwhere references to base chunks are maintained in the file description.These figures are discussed in conjunction with one another toillustrate an example application (FIG. 4) of the general process (FIG.7).

In one example embodiment, the garbage collection process utilizes deltareference information stored in the file description also referred toherein as the file recipe. The file recipe 401 includes a list of datachunks (e.g., X, Y and Z) that form the file where each data chunk isidentified by an address, fingerprint or similar information. In thisembodiment, along with the identification information for eachparticular data chunk, a delta reference is included where the datachunk has been delta encoded. For example, the data chunk X is stored asa delta encoding that utilizes a base chunk A and the file descriptionincludes a reference to A, labeled refA.

During the garbage collection process, the garbage collection moduletraverses or ‘walks’ each file recipe by selecting a next file recipe toprocess (Block 701) and traversing it before or in parallel with otherfile recipes. The process traverses a file recipe by selecting the nextdata chunk identifier in the file recipe (Block 703) and adding the datachunk identifier to a live chunk list (Block 705), then adding any deltareferences for the data chunk to the live chunk set (Block 707). In theillustrated example, the file recipe includes references to data chunksX, Y and Z, which are each added to the live chunk set. The data chunk Xis delta encoded and the delta reference is included with the filerecipe information. The illustration shows that the reference to thestored data chunk X in the data storage unit references the base chunkA. The process continues until all data chunks in a file recipe havebeen examined (Block 709) and all file recipes have been traversed(Block 711).

The live chunk set is utilized, in the example embodiment, to copy or‘sweep’ the live data chunks to a new location thereby freeing the priordata storage location (Block 713). The new storage location can utilizefewer resources if there were dead data chunks that were not copiedforward with the live data chunks. This embodiment is presented by wayof example and not limitation. One skilled in the art would understandthat this embodiment can be used in combination with the otherembodiments described herein above and below.

FIG. 5 is a diagram of one embodiment of a garbage collection processwhere references to base chunks are maintained in the metadata of a datachunk. FIG. 8 is a flowchart of one embodiment of a garbage collectionprocess where references to base chunks are maintained with the datachunk metadata. These figures are discussed in conjunction with oneanother to illustrate an example application (FIG. 5) of the generalprocess (FIG. 8). In one example embodiment, the garbage collectionprocess utilizes delta reference information stored in the data chunkmetadata. The file recipe 501 includes a list of data chunks (e.g., X, Yand Z) that form the file where each data chunk is identified by anaddress, fingerprint or similar information. The file recipe may or maynot indicate whether any data chunk is delta encoded.

During the garbage collection process, the garbage collection moduletraverses or ‘walks’ each file recipe by selecting a next file recipe toprocess (Block 801) and traversing it before or in parallel with otherfile recipes. The process traverses a file recipe by selecting the nextdata chunk identifier in the file recipe (Block 803) and adding the datachunk identifier to a live chunk list (Block 805). In the illustratedexample, the file recipe includes references to data chunks X, Y and Z,which are each added to a live chunk set. In this example, the datachunk X is delta encoded and the delta reference is included with thedata chunk metadata. In one embodiment, when the file recipe istraversed the metadata for each data chunk is examined to identifywhether a delta reference is included when the data chunk has been deltaencoded. For example, the data chunk X is stored as a delta encodingthat utilizes a base chunk A. The illustration shows that the referenceto the stored data chunk X in the data storage unit references the basechunk A. This delta reference can then added to the live chunk set. Inanother embodiment, described below the delta reference is determinedduring the copy phase. The process continues until all data chunks in afile recipe have been examined (Block 807) and all file recipes havebeen traversed (Block 809).

The live chunk set is utilized, in the example embodiment, to copy or‘sweep’ the live data chunks to a new location thereby freeing the priordata storage location. The new storage location can utilize fewerresources if there were dead data chunks that were not copied forwardwith the live data chunks. In the embodiment where the delta referenceis determined prior to the copy, the garbage collection process iscomplete.

In another embodiment, the delta-reference information is read duringthe access of the data chunk (Block 811) during the copy of theinformation to the new location. The delta reference information fromthe data chunk (e.g., the meta-data of the data chunk) is at that timeexamined and any base chunks that are not part of the live chunk set areadded to be copied as well (Block 813). In one embodiment, in the accessphase, the delta references can be collected in a separate traversal ofthe data chunks from the copying traversal. For example, all containerscan be traversed to identify all delta references in each containerincluding those that reference a data chunk in another container. Thisavoids a scenario where a data chunk in a particular container appearsto be dead, because no references to the data chunk were found in thecontainer (assuming a container by container traversal). However, areference to the data chunk could exist in another container.

The process of copying the data forward continues until all of the datachunks listed in the lives chunk set are copied to the new location(Block 815) including those delta references added during this process(i.e., encompassing single and multiple traversal embodiments). Thisembodiment is presented by way of example and not limitation. Oneskilled in the art would understand that this embodiment can be used incombination with the other embodiments described herein above and below.

FIG. 6 is a diagram of one embodiment of a garbage collection processwhere references to base chunks are maintained in an index. FIG. 9 is aflowchart of one embodiment of a garbage collection process wherereferences to base chunks are maintained in the index. These figures arediscussed in conjunction with one another to illustrate an exampleapplication (FIG. 6) of the general process (FIG. 9).

In one example embodiment, the garbage collection process utilizes deltareference information stored in the index. The file recipe 601 includesa list of data chunks (e.g., X, Y and Z) that form the file where eachdata chunk is identified by an address, fingerprint or similarinformation. The file recipe may or may not indicate whether any datachunk is delta encoded. The index 603 contains a set of entries thatcorrelate each data chunk with a location in a data storage unit. In oneembodiment, a set of entries (e.g., entry 605) in the index 603 includedelta reference information. In the illustrated example, entry 605 isfor data chunk X and includes its delta reference refA.

During the garbage collection process, the garbage collection moduletraverses or ‘walks’ each file recipe. A next file recipe is selected toprocess (Block 901) and it is traversed serially or in parallel withregard to other file recipes. The process traverses a file recipe byselecting the next data chunk identifier in the file recipe (Block 903)and adding the data chunk identifier to a live chunk list (Block 905).In the illustrated example, the file recipe includes references to datachunks X, Y and Z, which are each added to a live chunk set. In thisexample, the data chunk X is delta encoded and the delta reference isincluded with the index entry data. In one embodiment, when the filerecipe is traversed the index for each data chunk is examined toidentify whether a delta reference is included when the data chunk hasbeen delta encoded (Block 913). For example, the data chunk X is storedas a delta encoding that utilizes a base chunk A. The illustration showsthat the reference to the stored data chunk X in the data storage unitreferences the base chunk A. This delta reference can be found in theindex then added to the live chunk set. In another embodiment, the indexis traversed independently and after all file recipes have beenenumerated, and each delta reference associated with a live chunk foundtherein is added to the live chunk set. In the chunk by chunk indexchecking embodiment, the process continues until all data chunks in afile recipe have been examined (Block 907) and all file recipes havebeen traversed (Block 911). In the independent index traversalembodiment, the fingerprint index is traversed to index match livechunks and to add delta referenced base chunks to the live chunk setafter file recipe traversal completes (Block 907) and all file recipeshave been traversed (Block 911).

The live chunk set is utilized, in the example embodiment, to copy orsweep the live data chunks to a new location thereby freeing the priordata storage location (Block 915). The new storage location can utilizefewer resources if there were dead data chunks that were not copiedforward with the live data chunks. This embodiment is presented by wayof example and not limitation. One skilled in the art would understandthat this embodiment can be used in combination with the otherembodiments described herein above and below.

FIG. 10 is a flowchart of one embodiment of a sanitization process. Inone embodiment, a sanitization process can be used in combination withthe garbage collection process. In other embodiments, the sanitizationprocess is independent of the garbage collection process. The processcan be initiated during or after a garbage collection process, inresponse to the deletions or modification of a sensitive file or insimilar circumstances (Block 1001). An identifier for a deleted file canbe provided as an input or similarly received by the sanitizationprocess (Block 1003).

The file descriptor can then be traversed to identify references to datachunks that serve as base chunks for delta encoded data chunks (Block1005). It is assumed that the deletion process has removed any datachunks that do not serve as base chunks. Due to the sensitiveinformation or rules governing a file it may be necessary to also ensurethat all data chunks associated are deleted, a goal that could befrustrated by delta encoding schemes causing base chunks in the file tobe preserved. Each of the data chunks referencing the base chunk in thedeleted file (i.e., referencing a dead base chunk) can be decompressedusing the base chunk thereby reversing the delta encoding of thereferencing data chunk (Block 1007). The base chunk of the deleted filecan then be safely deleted without loss of data to other files and datachunks referencing the data chunks of the deleted file as base chunks(Block 1009).

FIG. 11 is a flowchart of one embodiment of a data chunk pre-fetchingprocess. In one embodiment, a pre-fetching process is utilized to load aset of data chunks into a cache or similar storage location with fasteraccess times in anticipation of utilization of at least some of the datain the set of pre-fetched data chunks. In one embodiment, this processis executed during a delta compression process as a data chunk isselected from a set of data chunks to be delta compressed and a set ofalready stored data chunks are retrieved for possible use as base chunksin the delta compression process.

The process can select and process data chunks serially or in parallel.The next data chunk to be delta compressed is selected (Block 1101). Thedata chunk is processed to determine a set of matching criteria such asa sketch or similar identifier to enable identification of candidatesfor serving as a base chunk in the delta compression process (Block1103). The matching criteria are then utilized to search for the set ofcandidates for serving as a base chunk using metadata stored in an indexof the stored data chunks or similar resource (Block 1105). In oneembodiment, a subset of the index stored in the working memory or cacheis searched and other portions of the index stored are loaded into thefaster access working memory or cache in accordance with processes andsystems discussed herein above and below. A matching data chunk or setof data chunks are loaded from the data storage unit and one is selectedto serve as the base chunk (Block 1107).

The data chunk selected to serve as the base chunk is then analyzed todetermine a set of locality data for the base chunk (Block 1109). Thelocality data can be any type of data utilized to identify similar datato be pre-fetched. The locality data can be location data (e.g.,addresses or address ranges, container information, data storage unitinformation, data stream location information or similar informationabout the storage location of the base chunk. The locality data can alsoencompass information relating the characteristics of the base chunkincluding the type, organization or form of data stored in the chunks,temporal information related to the chunk (e.g., chunks written atapproximately the same time), file locality (e.g., chunks written inpreceding versions of a file or similar files), co-location (e.g., inthe same container or similar data structure) or similar information.The locality data can then be utilize to search for and retrieve similardata chunks having matching or similar locality data (Block 1111). Thisprocess can be in-line or asynchronous with the general deltacompression process. The data chunks with matching or similar localitydata can then be loaded into a cache or similar storage structure with afast access time (Block 1113). In one embodiment, the data chunks areloaded based on a set of references to the data chunks held in thequeue, where the queue holds chunks or references to chunks that arebeing processed by any step described above or below. The availabilityof these data blocks increases the probability that subsequent datablocks to be delta compressed will find matching or similar data blocksin the cache rather than a slower data storage for use as base chunks indelta compression. The assumption is that subsequent blocks to be deltacompressed with find similar matches as preceding data blocks, which arelikely to be located proximate to the match of the preceding data block.In one embodiment, the pre-fetch process analyzes each data block forpotential pre-fetch. In other embodiments, the process only evaluatesperiodically or in response to a cache miss for a data block.

FIG. 12 is a flowchart of one embodiment of a data chunk selectionprocess where candidates are ranked by sketches. In one embodiment, thedelta compression system can factor in both the similarity and thelocation of a set of candidate data chunks to serve as a base chunk fora delta encoding of a data chunk. The process can be initiated inresponse to receiving data to delta encode, as a periodic process orunder similar circumstances.

The process selects a next data chunk to be delta encoded (Block 1201).The selected data chunk is then analyzed to determine a set ofsimilarity data by execution of a function such as a resemblance hash orsimilar function (Block 1203). The result, such as a sketch, can then beutilized to identify a set of possible candidate data blocks to serve asa base chunk (Block 1205) by using the sketch as a query against theindex or similar data structure. The similarity or degree of match withthe sketch can be based on a number of features within the sketch thatmatch or based on similar measures of a degree of a match, which can beused to rank the candidates (Block 1207). The search for candidates canutilize an index or similar data structure that stores a sketch for eachdata block and enables a quick comparison of the degree of similaritybetween data blocks in the system.

Each of the candidates with at least one matching feature or degree ofmatching features can be ranked for possible consideration as a basechunk in the order of their degree of similarity with the selected datachunk (Block 1207). More than one of the data chunks could have the samedegree of matching creating a tie as to which data block has a closestmatch with the data chunk to be encoded. In this case, location statusinformation for each of the candidate data chunks can be considered totie break. The location status information includes information aboutthe factors that affect the speed at which a data chunk can be retrievedwith the various statuses presenting information such as location of thedata in a cache or a data storage unit, compression status and similarinformation. A candidate data chunk can then be selected based on thecombination of the similarity ranking and a tie-break based on locationstatus (Block 1209). The delta compression is then executed with theselected candidate (Block 1211), then committed to storage ortransmitted. This selection process prioritizes compression, but wherecompression would be approximately the same elects to utilize a basechunk that can be loaded faster.

In a further embodiment, the process can monitor for whether anycandidate data chunk meets a defined threshold for similarity, in whichcase the process can be halted and the candidate meeting the thresholdcan be selected for use as a base chunk. Such thresholds can also beused in combination with the dynamic candidate selection discussedbelow.

FIG. 13 is a flowchart of one embodiment of a data chunk selectionprocess where candidates are ranked by location status. In oneembodiment, the data chunk selection process to identify base chunks isprioritized for speed over similarity that would provide higher levelsof compression.

In this embodiment, the process selects a next data chunk to deltacompress (Block 1301). The selected data chunk is then analyzed todetermine a set of similarity data by execution of a function such as aresemblance hash or similar function (Block 1303). The result, such as asketch, can then be utilized to identify a set of possible candidatedata blocks to serve as a base chunk (Block 1305). The similarity ordegree of match with the sketch can be based on a number of featureswithin the sketch that match or based on similar measures of a degree ofa match. The search for candidates can utilize an index or similar datastructure that stores a sketch for each data block and enables a quickcomparison of the degree of similarity between data blocks in thesystem.

Location status information for each candidate base chunk with at leastone similar feature or a minimum degree of similarity can be determined.The location status information includes information about the factorsthat affect the speed at which a data chunk can be retrieved with thevarious statuses presenting information such as location of the data ina cache or a data storage unit, compression status and similarinformation. Each of the candidates with at least one matching featureor degree of matching features can be ranked for possible considerationas a base chunk in the order of their proximity in location status orfastest access time based on the location status (Block 1307). More thanone of the data chunks could have the same proximity or access timecreating a tie as to which data chunk can be loaded the fastest to beused for encoding. In the event of a tie, the degree of similarity tothe selected data chunk can be utilized as a tie breaker (Block 1309). Acandidate data chunk can then be selected based on the combination ofthe location status and similarity ranking and a tie-break based onsimilarity ranking (Block 1309). The delta compression is then executedwith the selected candidate (Block 1311), then committed to storage ortransmitted. This selection process prioritizes speed, but where speedwould be approximately the same elects to utilize a base chunk thatprovides a better level of compression due to its higher similarity tothe selected data chunk.

In a further embodiment, the process can monitor for whether anycandidate data chunk meets a defined threshold for location status(e.g., an access time), in which case the process can be halted and thecandidate meeting the threshold can be selected for use as a base chunk.Such thresholds can also be used in combination with the dynamiccandidate selection discussed below.

FIG. 14 is a flowchart of one embodiment of a process for dynamiccandidate selection. In one embodiment, the process adjusts depending onresource availability to maintain a throughput to manage the incomingdata chunks to be delta compressed. The process selects a next datachunk to delta compress (Block 1401). The process can utilize anyselection mechanism for identifying candidate data chunks to serve as abase chunk such as the selection mechanisms described above in regard toFIGS. 12 and 13. The currently selected data chunk selection mechanismfor identifying a base chunk is applied (Block 1403). This selectedcandidate can then be used for the delta compression of the selecteddata chunk. The process then continually monitors the availableresources and throughput of the delta compression process (Block 1405).

The resource utilization and throughput is analyzed to determine whethera change in the current data chunk candidate selection process needs tobe adjusted (Block 1407). This can be determined by checking a number ofdata chunks to be processed (e.g., in a queue of such data chunks),monitoring throughput rates and comparing with input rates and throughsimilar direct and derived metrics. If the amount of resources availableis high (and underutilized) and/or if the throughput is high, then theselection mechanism can be changed to utilize a selection mechanism thatprioritizes similarity between the selected data chunk and the set ofcandidates to increase the degree of compression (block 1411). There maybe any number of possible selection mechanisms available to be utilizedin which case the dynamic process can incrementally adjust the selectionmechanism until one end of the spectrum of selection mechanisms isreached.

If the resources available and/or throughput are low, the currentselection mechanism can be adjusted to increase throughput by movingtoward a selection mechanism that emphasizes location status andthroughput over compression (Block 1413). In this regard, it is possibleto completely disable delta compression if the resources are too low orthroughput has fallen too far to enable the delta compression system toprocess the incoming data chunks. If however, the delta compressionprocess is using a moderate amount of resources and/or generating amoderate output, the current selection mechanism may be maintained(Block 1409).

FIG. 15 is a flowchart of one embodiment of a process for sub-chunkdeduplication. FIGS. 16 and 17 are diagrams of each stage of thesub-chunk deduplication process and are discussed in conjunction withthe general process of FIG. 15. In one embodiment, sub-chunks and themetadata associated with them can be created at the time that datachunks are defined in the delta compression system. In otherembodiments, the sub-chunks and their metadata are defined at a laterpoint in the process or in a separate process. The process starts byselection of a next data chunk to process (Block 1501). In theillustrated example of FIG. 16, the selected data chunk is labeled B*. Asketch is made of the selected data chunk using a resemblance hash orsimilar process (Block 1503). The sketch is utilized to identify similardata chunks with matching or similar sketches (Block 1505). In theillustrated example of FIG. 16, the metadata of the chunks includes asketch that is used to identify that data chunk B is similar to B*. Themetadata also contains the fingerprints of the sub-chunks of data chunkB. The process loads the sub-chunk metadata of the similar chunk, in theexample data chunk B (Block 1507). This metadata can be retrieved fromthe metadata of a container of the chunk.

The fingerprints of each of the sub-chunks are compared to fingerprintsof the sub-chunks of the selected data chunk, in the example data chunksB* (Block 1509). The sub-chunks with matching fingerprints are encodedas references to the matching sub-chunks of the matching data chunkusing the fingerprints of the matching sub-chunks (Block 1511). Thosesub-chunks where there is not a match are stored as-is or delta encodedwith a similar sub-chunk serving as the base sub-chunk. The matching andencoding is illustrated in FIG. 17 showing the organization of datachunk B and data chunk B* as well as the encoding of B*. In the examplethe sub-chunk SFP4 has no direct match because it contains changed data.Sub-chunks do not have to match in order of the base chunk's sub-chunksand can have varied order. Similarly, the size and organization of thesub-chunks can be fixed or variable and overlapping of sub-chunks ispermissible in some embodiments. One skilled in the art would understandthat any configuration of sub-chunks is compatible with the process andstructures described herein.

Some portions of the preceding detailed descriptions have been presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as those set forth in the claims below, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the invention also relate to an apparatus for performingthe operations herein. Such a computer program is stored in anon-transitory computer readable medium. A machine-readable mediumincludes any mechanism for storing information in a form readable by amachine (e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices).

The processes or methods depicted in the preceding figures may beperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software (e.g., embodied on a non-transitorycomputer readable medium), or a combination of both. Although theprocesses or methods are described above in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed may be performed in a different order. Moreover, someoperations may be performed in parallel rather than sequentially.

Embodiments of the present invention are not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof embodiments of the invention as described herein.

In the foregoing specification, embodiments of the invention have beendescribed with reference to specific exemplary embodiments thereof. Itwill be evident that various modifications may be made thereto withoutdeparting from the broader spirit and scope of the invention as setforth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

What is claimed is:
 1. A computer-implemented method for improving efficiency in a delta compression process in a data storage system, the method comprising: selecting a data chunk to delta compress; generating a sketch for the selected data chunk; searching for a set of candidate data chunks using the sketch; ranking the set of candidate data chunks by degree of sketch matching; tie-breaking the set of candidate data chunks, where the set of candidate data chunks has an equal degree of sketch matching, including ranking a candidate data chunk stored in a cache higher than a candidate data chunk stored on a storage device, and, ranking a decompressed candidate data chunk higher than a compressed candidate data chunk having a same location status as the decompressed candidate data chunk; delta compressing the selected data chunk with a selected candidate data chunk of the ranked set of candidate data chunks; and storing the delta compressed selected data chunk in the data storage system.
 2. The method of claim 1, wherein generating the sketch further comprises: generating a set of features for the selected data chunk.
 3. The method of claim 2, wherein searching for the set of candidate data chunks further comprises: searching an index of data chunks to identify the set of candidate data chunks by comparing features in the index with the set of features of the selected data chunk.
 4. The method of claim 1, wherein ranking the set of candidates data chunks by degree of sketch matching further comprises: ranking the set of candidates by a number of matching features between a candidate and the selected data chunk.
 5. The method of claim 1, wherein the location status data indicates the location and status of the candidate data chunk as in any one of a compressed in a cache status, a decompressed in a cache status, or a compressed in a data storage status.
 6. A computer-implemented method for improving efficiency in a delta compression process in a data storage system, the method comprising: selecting a data chunk to delta compress; generating a sketch for the selected data chunk; searching for a set of candidate data chunks using the sketch; ranking the set of candidate data chunks by degree of sketch matching; tie-breaking the set of candidate data chunks, where the set of candidate data chunks has an equal degree of sketch matching, using location status data for each candidate data chunk, wherein the location status data indicates one or more of the location of a corresponding candidate data chunk in a set of memory devices or a compression status of the corresponding candidate data chunk, wherein the location status data indicates the location and status of the candidate data chunk as in any one of a compressed in a cache status, a decompressed in a cache status, or a compressed in a data storage status, wherein tie-breaking the set of candidate data chunks using location status data for each candidate prefers a decompressed in a cache status over a compressed in a cache status, and wherein tie-breaking the set of candidate data chunks using location status data for each candidate prefers a compressed in a cache status over a compressed in a data storage status; delta compressing the selected data chunk with a selected candidate data chunk; and storing the delta compressed selected data chunk in the data storage system.
 7. The method of claim 1, wherein tie-breaking the set of candidate data chunks using location status data for each candidate orders the set of candidate data chunks based on access times to retrieve each candidate data chunk.
 8. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform a method, the method for improving efficiency in a delta compression process, the method comprising: selecting a data chunk to delta compress; generating a sketch for the selected data chunk; searching for a set of candidate data chunks using the sketch; ranking the set of candidate data chunks by degree of sketch matching; tie-breaking the set of candidate data chunks, where the set of candidate data chunks has an equal degree of sketch matching, including ranking a candidate data chunk stored in a cache higher than a candidate data chunk stored on a storage device, and, ranking a decompressed candidate data chunk higher than a compressed candidate data chunk having a same location status as the decompressed candidate data chunk; delta compressing the selected data chunk with a selected candidate data chunk of the ranked candidate data chunks; and storing the delta compressed selected data chunk in a data storage system.
 9. The non-transitory computer-readable storage medium of claim 8, wherein generating the sketch further comprises: generating a set of features for the selected data chunk.
 10. The non-transitory computer-readable storage medium of claim 9, wherein searching for the set of candidate data chunks further comprises: searching an index of data chunks to identify the set of candidate data chunks by comparing features in the index with the set of features of the selected data chunk.
 11. The non-transitory computer-readable storage medium of claim 8, wherein ranking the set of candidates data chunks by degree of sketch matching further comprises: ranking the set of candidates by a number of matching features between a candidate and the selected data chunk.
 12. The non-transitory computer-readable storage medium of claim 8, wherein the location status data indicates the location and status of the candidate data chunk as in any one of a compressed in a cache status, a decompressed in a cache status, or a compressed in a data storage status.
 13. The non-transitory computer-readable storage medium of claim 8, wherein tiebreaking the set of candidate data chunks using location status data for each candidate orders the set of candidate data chunks based on access times to retrieve each candidate data chunk.
 14. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform a method, the method for improving efficiency in a delta compression process, the method comprising: selecting a data chunk to delta compress; generating a sketch for the selected data chunk; searching for a set of candidate data chunks using the sketch; ranking the set of candidate data chunks by degree of sketch matching; tie-breaking the set of candidate data chunks, where the set of candidate data chunks has an equal degree of sketch matching, using location status data for each candidate data chunk, wherein the location status data indicates one or more of the location of a corresponding candidate data chunk in a set of memory units or a compression status of the corresponding candidate data chunk, wherein the location status data indicates the location and status of the candidate data chunk as in any one of a compressed in a cache status, a decompressed in a cache status, or a compressed in a data storage status, wherein tiebreaking the set of candidate data chunks using location status data for each candidate prefers a decompressed in a cache status over a compressed in a cache status, wherein tiebreaking the set of candidates data chunks using location status data for each candidate prefers a compressed in a cache status over a compressed in a data storage status; delta compressing the selected data chunk with a selected candidate data chunk; and storing the delta compressed selected data chunk in a data storage system.
 15. A delta compression system, comprising: a delta processing module to delta compress a first set of data chunks; and a cache to store a second set of data chunks; a data storage system to store a third set of data chunks; a preferential selection module coupled to the cache, data storage system and delta processing module, the preferential selection module to identify a candidate to serve as a base chunk for delta compression by ranking a set of candidate base chunks from the second set of data chunks and third set of data chunks by similarity to a data chunk from the first set of data chunks then tie-breaking a ranked set of candidate data chunks, where the set of candidate data chunks has an equal degree of sketch matching, tie-breaking comprises ranking a candidate data chunk stored in a cache higher than a candidate data chunk stored on a storage device, and, ranking a decompressed candidate data chunk higher than a compressed candidate data chunk having a same location status as the decompressed candidate data chunk; the delta compression module configured to: select a data chunk to compress from the first set of data chunks; delta compress the selected data chunk with the identified candidate to serve as a base chunk for delta compression; and store the delta compressed selected data chunk in the data storage system.
 16. The delta compression system of claim 15, wherein the location status data indicates the location and status of the candidate data chunk as in any one of a compressed in a cache status, a decompressed in a cache status, or a compressed in a data storage status.
 17. A delta compression system comprising: a delta processing module to delta compress a first set of data chunks; and a cache to store a second set of data chunks; a data storage system to store a third set of data chunks; a preferential selection module coupled to the cache, data storage system and delta processing module, the preferential selection module to identify a candidate to serve as a base chunk for delta compression by ranking a set of candidate base chunks from the second set of data chunks and third set of data chunks by similarity to a data chunk from the first set of data chunks then tie-breaking a ranked set of candidate data chunks, where the set of candidate data chunks has an equal degree of sketch matching, using location status information, wherein the location status information indicates one or more of the location of a corresponding candidate data chunk in a set of memory devices or a compression status of the corresponding candidate data chunk, wherein the location status data indicates the location and status of the candidate data chunk as in any one of a compressed in a cache status, a decompressed in a cache status, or a compressed in a data storage status, wherein tie-breaking the set of candidate data chunks using location status data for each candidate prefers a decompressed in a cache status over a compressed in a cache status, wherein tie-breaking the set of candidate data chunks using location status data for each candidate prefers a compressed in a cache status over a compressed in a data storage status. 